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(57) Abstract 

An animation system comprises a scheduler (1) which sequentially triggers a conversation engine (3), an events manager (4) and a 
renderer (5) during each frame period. Each of the above modules (3 to 5) reads previously written data from and writes data to a data 
structure (7) which contains a dynamically evolving set of data including hierarchical object data and event data which determines the 
behaviour of the objects. The objects include avatars which interact with the game player in a 3D environment. The animation system is 
capable of managing a non-linear interactive storyline such as that used in the playing of an interactive computer game. 
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Animation System and Method 

The present invention relates to an apparatus for managing and generating animation and to 
a method of generating an animation sequence. 

It is envisaged that such apparatus and method will be useful in the film industry and also in 
the field of computer games for example. 

The invention relates particularly but not exclusively to photo-realistic three-dimensional 
animation, i.e. to animation which represents three-dimensional movement in a three- 
dimensional environment scene on a display in a manner approaching photographic realism. 

The digital effects industries have produced powerful software packages that allow 
technically skilled computer artists to create such animation. However such packages are 
generally too complex for non-technical people with traditional entertainment production 
talents. Particularly in the computer games industry, this has tended to result in products 
which lack the creative spark of traditional entertainment media such as motion pictures, 
television or theatre. 

An object of the present invention is to provide an animation system and method which 
overcomes or alleviates the above problems. 

In one aspect the invention provides a system for managing animation comprising a 
processor arranged to process a dynamically evolving data set including object data and 
event data, said object data including character data representing avatars, sequencing means 
arranged to read data from and write data to said dynamically evolving data set in a 
programmed sequence, said sequence of reading and writing taking place within a frame 
period and said written data being generated from said read data by stored interactions 
between said object data and event data, and output means arranged to generate from said 
dynamically evolving data set a displayable animated output 

The system is therefore advantageously capable of providing a real-time animated output 
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which enables the managing of a non-linear interactive storyline, e.g. for the playing of an 
interactive computer game. In such an interactive storyline, which is also known as a 
branching script, the user (or player) can determine how the storyline develops by making 
selected choices during its progression. The system is also capable of managing a linear 
storyline, such as is used with a non-interactive computer game, where there are no 
significant time constraints for the generation of each frame of the storyline. 

A method of generating avatars (artificially-generated animated figures in electronic format) 
suitable for photo-realistic animation is described in our co-pending patent application GB 
9610212.4 (a copy of which is filed herewith as an appendix). 

Preferably said data set includes stored events, each event comprising a set of one or more 
preconditions associated with a set of one or more results, and the system includes an events 
manager controlled by said sequencing means which is arranged to detect said set of 
preconditions and, on detection of said set of preconditions, to generate said set of results, 
said output means being responsive to said results to generate said displayable animated 



For example an event may comprise the set of preconditions: 

Bob is at room 1 
Fred is clicked 
Fred is at room 2 

and the set of results: 

Move Bob to room 3. 

The above result could for example be implemented by a stored animation routine which 
would move the avatar ABob@ to room 3. 

Accordingly the system preferably comprises animation routines arranged to be triggered by 
said generated sets of results, said output means being arranged to run said triggered 
animation routines to generate said displayable animated output. 

Preferably the system comprises rendering means arranged to render a scene composed of 
objects associated with said generated sets of results. 



output. 
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Preferably the object data represent a three-dimensional world. 

In another aspect, the invention provides a method of generating an animation sequence 
comprising the steps of generating a dynamically evolving data set including object data 
and event data, said object data including character data representing avatars, reading data 
from and writing data to said dynamically evolving data set in a programmed sequence, aid 
sequence of reading and writing taking place within a frame period and said written data 
being generated from said read data by stored interactions between said object data and 
event data, and generating a displayable animated output from said dynamically evolving 
data set. 

Preferably a program containing stored input instructions in a high level language and 
governing the animation sequence is run and is arranged to modify said dynamically 
evolving data set. 

The invention also provides a computer programmed with a computer game containing an 
animation sequence generated by a method in accordance with the above aspect of the 
invention. 

Preferred features are defined in the dependent claims. 

Preferred embodiments of the invention are described below by way of example only with 
reference to Figures 1 to 4 of the accompanying drawings, wherein: 

Figure 1 is a block diagram showing the architecture of one embodiment of the invention 
from the standpoint of running a previously created animated game; 

Figure 2 is a block diagram showing a hierarchy of object types which can be used in the 
above embodiment; 

Figure 3 is a block diagram of one example of a set of objects which can be used in the 
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above embodiment, and 

Figure 4 is a block diagram showing the architecture of the above embodiment from the 
standpoint of the creator of an animated computer game. 

The above embodiment is based on a Pentium® - equipped personal computer running the 
Windows95® operating system as indicated by reference numeral 2 (Figure 1). 

Referring to Figure 1, the arrangement comprises a conversation engine 3, an events 
manager 4 and a renderer 5 which are sampled sequentially in that order during each frame 
period of 1/12 second by a scheduler 1. Conversation engine 3, events manager 4 and 
renderer 5 are all software modules in this embodiment but could in principle be 
implemented by dedicated hardware. 

In principle the sampling order could be modified by an output of any of the above modules 
3 to 5 and accordingly there is provision for two-way data flow (indicated throughout by 
double-headed arrows) between these modules and scheduler 1. There is also provision for 
two-way data flow between each of the above modules and a data structure block 7, which 
contains a dynamically evolving data set including object data and event data, the object 
data including character data representing avatars. 

In use, each of the above modules 3 to 5 reads appropriate portions of the above data set in 
response to a signal from scheduler 1 , processes the above data and then writes the result in 
the data set. Each module 3 to 5 reads and writes in the above fashion during each frame. 
Hence events manager 4 reads what conversation engine 3 has written at the beginning of 
the frame and renderer 5 subsequently reads what events manager 4 has written in the same 
frame, generates a rendered output (which is sent to the direct X, direct draw and direct 3D 
modules of Windows 95@ and then output through a port of the computer) before the frame 
ends. The above sequence is then repeated for subsequent frames. 

The conversation engine 3 controls responses of the avatars (Non-Player Characters) to he 
conversation of the (human) player of the game and has provision for two-way data flow 



BNSDOCID: <WO 9835320A1_I_> 





. WO 98/35320 



m 



PCT/GB98/00372 



with the direct sound component of Windows 95®, enabling speech input and sound output 
via an appropriate sound card, microphone and speakers) (not shown). It can generate, and 
write to the data structure block 7, the precondition components of events which are 
handled by the events manager 4. 

In order to do this, the conversation engine 3 operates on a conversational Anet@ of 
sentence units, nodes in the net representing pauses in a conversation, where the participants 
pause, and consider what to say next. The branches from a node represent the possible 
sentences which could be spoken in response to the sentence which has led to that node 
(having just been spoken). Furthermore the sentences in the net model not only what is 
spoken but also events (particularly preconditions to which the events manager 4 is 
responsive). 

It is possible to control any aspect of the game interaction as a result of the way a 
conversation path through the above net is followed. 

All conversations are stored in a comma delimited file, which is loaded into the system at 
the start of a game. The first line of data details the conversation's parameters, specifically: 



Conversation ID (string) Text description 



Number of sentences 



Number of participants Number of results 
First speaker 
n participants 



n results 



Each of the following lines detail a sentence of the conversation: 



Sentence ID (integer) 



Conversation ID 



Cue 



Text 
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Current node 



Next node 



Nextspeaker 



Number of results 



n Results 



When a conversation is due to start, the list of participants is checked to ensure they are at 
the same location, aborting if they are not. 

The 'node 1 value is set to zero, (this identifies where the conversation is logically), and the 
'Current speaker 1 is set from the conversation data. 

The list of sentences is searched for all sentences with the correct conversation ID and a 
'current node' value equal to the 'node' value. If the current speaker is the player of the 
game, a menu of the sentence cues is offered, or else a sentence is randomly picked. The 
sentence is then played by a text-to-speech system. Finally, the 'current speaker 1 and node 1 
values are updated, and the sentence's results are processed. This loop is then repeated. 

When a conversation ends, the conversation's results are played before the last sentence 1 
results, as this allows for varying conversations to be started by different end sentences. 

As noted above, the events manager 4 responds to certain preconditions created, for 
example, by the conversation engine 3 by generating certain results. An Aevent@ is defined 
as a set of preconditions associated with a set of results and the function of the events 
manager 4 is essentially to recognise the appropriate sets of preconditions and to generate 
the appropriate sets of results, which are written to the data structure block 7. 

A precondition is some sort of test, to which the result can be TRUE or FALSE. A result is 
any command which any of the various engines 3 to 5 understand. 
A typical event may look like this: 

EVENT I 

PRECONDITIONS > RESULTS 
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BOB IS AT ROOM1 



MOVE BOB TO ROOM3 



FRED IS CLICKED 
FRED IS AT ROOM2 

Each event is associated with an object in the world. When an object receives an 'update* 
command, it will examine the list of events associated with it, and execute the event with all 
TRUE preconditions, and with the highest priority. The execution of an event's results is 
known as firing. 

This association between events and object does have one important implication: currently, 
only objects within a room occupied by a player=s avatar are updated. If an event needed 
happened in another room, it would only happen when the player arrived. 

For example: in a twist to the plot, the head of marketing decides to dynamite the MD's 
office. The dynamite object has an 'explode 1 event, with a precondition that it has been in 
the room for 10 seconds. Until the player is in the MD's office, the dynamite will not be 
updated. Potentially a lethal situation for the player. 
The following information defines an event: 



An unique integer between 1 and 999,999,999. This is currently user defined, but could be 



A text description of what the event does, that is currently ignored and lost when the 
window is closed. 



If the user required an event that will repeat at timed intervals, such as the ticking of a 
clock, specify the number of 'update loops' which the event will wait for. It is important to 
check that the delete after fire box is not active, which could be done in software. 



ID 



automated. 



Description 



Frequency 
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Acceptable frequencies range from 0 to 64,000. Note that an event with a frequency of 0 
will try to fire on every update. 



An integer from 0 to 99999999, with 0 being the most urgent. If a number of events are 
ready to fire, the event with the lowest priority figure will fire. Currently, if a number of 
events have the same priority, then the first one found is fired. Ranges need to be 
determined for effective use of this feature. 



This tick box is activated if the event is expected to only fire once. 

An event can have any number of preconditions in its preconditions section. These 
determine if the event is able to fire. If there is more than one precondition in an event then 
they are treated with Boolean AND logic, i.e. they must all be true if the event is to fire. 
There could be other logical operations available, such as OR, NOT and XOR. If there are 



Priority 



Delete after fire 



no preconditions specified, then the event may always fire. 



The following preconditions are currently supported: 



[x] IS AT [y] 
True if object X's owner is Y. 
e.g. Bill IS AT Whitehouse 



TRUE 



True! 



[x] CLICKED 
True if object X has been right-clicked in the 3D view, 
e.g. Bill CLICKED 
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RELATION WITH [x] IS +/- [Z] 
True if player ! s relationship with person X is greater or less than Z. 
e.g. RELATION WITH Bill + 5 



EVENT [x] COMPLETED 
True if event X has finished firing, 
e.g. EVENT 2468 COMPLETED 



TIMEINROOM [x] GT [y] SECS 
True if object X has been at its current location for more than Y seconds, 
e.g. TIMEINROOM reception GT 10 SECS 

CLICKSINROOM GT [x] 
True if the user has right-clicked in the current room more then X times, 
e.g. CLICKSINROOM GT 2 

REMEMBERS [x] 
True if the player has been instructed to remember a string X. 
e.g.: REMEMBERS SLARTIBARTFART 



COLLIDED [what] 

This precondition will return TRUE if the given object has collided with something. It is 
then up to the gameplanner to decide what to do next, A good use would be to combine the 
precondition COLLIDED with the new RANDOMPLAY result to get various wincing 
"oohs" and "aahs" from the affected characters (objects) such as avatar heads. 



A user interface 6 is provided to enable two-way data flow to and from events manager 4. 

The Tenderer 5 reads the necessary data from block 7 and then generates a rendered video 
output. The renderer 5 can be as described in our co-pending application referred to above 
and incorporated herein by reference. 
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The described embodiment utilises objects which interact with the events handled by the 
events manager 4. Players, rooms, non-player characters, items of furniture, synthetic 
displays, and anything else held in the model of the game world are all objects. The system 
needs to know what is in the world, what each object is in terms of name, location, 
associated graphics and behaviours, and gameplan events linked to that object. 

The events manager 4 can poll each object to determine whether it has an associated event 
to execute, namely if it has an event to fire. As an event is fired, the events manager 4 routes 
the results to the appropriate engines 3 to 5 for execution or executes those results itself. 
The events manager 4 ensures that the correct action is carried out in the correct manner at 
the correct time by the correct object or engine. 

Each object has an associated script file which describes in detail the objects graphics, 
behavioural attributes, and initialisation and cleanup functionality. 

Objects are modelled with consideration to their complexity. More sophisticated object 
types inherit functionality from simpler types. 



Referring now to Figure 2, a hierarchy of object types is shown. The basic object type is an 
Aartefact@ 8. This divides into Acontainer@ 9 and camera 10 categories. A container 9 
could be a room 1 1 or a person 12, and the latter could be a player 13. 

Thus a person is represented as a container 9 plus any person-specific functionality. The 
container 9 in turn inherits much from artefact 8, notably the basic ability to have a name, a 
location, and be associated with gameplan events and 3D graphics. 

The container 9 adds the functionality of knowing how to store other objects from the 
gameworld within itself. Room & player have some common functionality and attributes 
and some distinctive functionality and attributes. For example only rooms have doors and 
this is an integral part of being a room. Despite this, both people and rooms can contain 
things (think of a person containing something as a person 'carrying something). So at this 
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point it is logical to have to separate objects for Room 11 and Person 12 both of which 
inherit the functionality of containers but redefine their own distinct functionality. 

Cameras 10 can be set anywhere in the 3D environment to capture a view of a scene. A 
stand-alone camera 10 may provide a view from vantage points; a moving camera be 
directed using movie commands; a POV camera may be fixed to a moving object such as a 
player for first-person or third-person tracking. Multiple cameras may be used; gameplan 
event results and movie commands allow complex cinematography. 

Referring now to Figure 3, which shows the structure of a group of objects, it will be seen 
that actual objects as well as classes of objects are stored as hierarchical trees. The root 
object 14 is an abstract object which is used to link a camera 15 and its associated light 16 
with a table 17 supporting a vase 18 which contains flowers 19. The links between these 
objects mean that they can behave as a group; for example if the table 17 moves the vase 18 
moves with it, as do the flowers 19. In fact, in this case the table 17 will always be 
stationary with respect to the depicted camera 15 (which in turn will be lit in an unchanging 
manner by light 16) but in principle the table can be viewed by another camera (not shown). 
A result from an event can be applied to the root object 14 and will then automatically 
affect all the objects 14 to 19 depending from it. In principle a result of an event could act 
on the vase 17, for example, to knock it off the table 17, in which case the flowers 19 would 
be knocked off too. 

Referring to Figure 4, which shows the preferred system from the point of view of the 
creator of the game (the gameplanner) the user populates the data structure 7 with objects 
and events via an interface 6 which preferably includes a graphical user interface with a 
pointing device as well as a keyboard. The objects are stored hierarchically as described 
above with reference to Figure 3 and the events held in the system represent the gameplan. 

Each object has one script file associated with it in addition to the object attributes entered 
into the object's property pages. Script files are held as text data separate from the program. 
They must have the same name as the object they refer to, they must reside in the SCRIPTS 
directory and must only be edited with a plain-text editor. 



BNSDOCID: <WO_9835320A1_I_> 



. WO 98/35320 




PCT/GB98/00372 



A script file is divided into several sections, with certain sections reserved to do certain 
things and custom sections designed by the gameplanners. The sections are as follows: 

Standard Script File Sections 

Script file sections may contain behaviour attributes, event results, movie commands 
or internal script control commands. Script file sections may be generic for all objects such 
as initialisation - describing initial settings, common for a particular type of object, e.g. 
wallcing for players, or specific to a particular individual object. 

The following script file sections are standard: 

[INIT] to [ENDINIT] 
Load graphics and models. 

[POSTINIT] to [ENDP OSTIN1T] 
Assign graphics to models. 

[DELETE] to [ENDDELETE] 
Delete graphics. 

[BLINK] to [ENDBLINK] 
Randomly called random movement. Facial movement is available as standard for players. 

Usage is optional. 

[STARTWALK] to [ENDSTARTWALK] 
Start a walking animation. 

[WALK] to [END WALK] 
Loop a walking animation. 
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[STOPWALK] to [ENDSTOPWALK] 
Stop a walking animation. 



Custom Script File Sections 

Custom sections may be designed for the object which are then referenced from within the 
script or from gameplan event results. This allows for the creation of object-specific 
behaviours and attributes under the control of the gameplan or object script. 

Script file sections may contain movie commands or event results, which allow for 3D 
manipulation of the environment or more complex gameplan related actions to be triggered 
from within a script. 

Scripts can be called from the gameplan by two gameplan event results: Playscript and 
Runscript. The former will execute all the lines of code between a specified start-marker 
and end-marker in one sweep. Runscript will execute the first line of a script, (presumed to 
be a SELECT), then execute one line per update. This allows flexible access to the 
gameplanner to the lowest level properties of an object during the game progression. 

Behaviour Attributes are set as flags set in the script file which specify attributes of object 
behaviour. Whilst the same effects could be duplicated with events, the repetition of 
creating these events would drive the writers to desperation. In cases of common default 
behaviours or available behaviours such as blinking, actual code has been provided within 
the system which can be triggered by the appropriate behaviour attribute flag in the script 
file, in this case TYPEBLINKER. 

TYPEBLINKER Player avatar has the ability to blink, and do so according to the initialised 
or currently set behavioural parameters. 

The following behavioural attribute flags are available: 
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AAFPLAYER 

Specifies the script playing object to be the player's object. 

ISSTATIC 

Specifies that the script playing object will not move, so only needs to be rendered once 

when in a third person point of view. 

TYPE BOUNCER 1 

An object of this type will randomly glide around the screen, bouncing off other objects and 
the sides of the screen. Implemented for screen savers. 

TYPE BLINKER1 [N] 

Specifies that the object will blink randomly. If no value is supplied then a default value of 

20 is used. The chance of blinking is 1 in N. Note that 'blinking 1 can be any random 
movement as defined by the actual section in the script file, such as raising of eyebrows, to 

the swinging of a tail. 

TYPE NOBLINKER1 
Stops an object from blinking. Same as TYPE B LINKER 1 0. 

FLOOR 

Specifies the object is the floor of a room. If the player collides with the floor, the collision 

is ignored. 

REMOVEFLOOR 
Removes the 'FLOOR 1 status of an object. 

Script control commands regulate the processing of the scripts. The following commands 
are available: 
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is 



RUNSCRIPT start end 



Removes the current script from the controlling objects list of active script parts, and adds 



ADDS CREPT start end 
Adds the new script part, as specified by the start and end identifiers to the controlling 

objects list of active script parts. 

KILLSCRIPT start 

Removes the script specified by the start identifier from the controlling objects list of active 

script parts. 



Repeats the specified command N times. Not compatible with command +. Executes 
command, then the command on the next line. Not compatible with [N], and should be 
used with caution when nearing the end of a script section. 

In order to simplify the game authoring process, a movie engine 20 is provided which is 
operated by movie commands from interface 6 and runs corresponding pre-stored 
animation sequences, which could have been captured previously from a motion capture 
device 22. Non-programmers can perform actions in 3D environments using English-like 
commands. The required movie commands can be executed from object script files or from 
event results as generated by the events manager described above. The motion capture can 
be as described in our co-pending earlier application referred to above. 

Furthermore, a lip synch engine 21 is provided which generates timing files from sampled 
speech (.WAV) files (which are in turn associated with sentences in the previously 
described conversation net) in response to certain types of sound waveform associated with 
specific lip movements such as the sounds A EE O OO and hard sounds such as T. These 
timing files are used to call appropriate lip animation routines. 



the new script part, as specified by the start and end identifiers. 



[n] command # 
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An alternative preferred embodiment of the present invention, which is described below, is 
similar to the above described embodiment except for the different way in which the events 
manager 4 operates. Accordingly, to prevent unnecessary repetition, only the differences 
between the two embodiments are described below. 

In the previous embodiment, the events manager 4 polls each object to determine whether 
any event associated with the object has been fired. However, in the alternative 
embodiment, the events manager 4 incorporates a state machine map (not shown) for each 
object to establish how each object is to react to changes in its environmental conditions. 

The state machine map establishes a set of different states in which the object can be. The 
object resides in one state at a time and moves to a different state by the action of an event 
on the object. Movement from one state to another is initiated by a predetermined change in 
a set of environmental conditions occurring locally to the object. Other non-predetermined 
changes in the local conditions have no effect on the object and the object remains in its 
current state. This is best illustrated by way of the following example: 

Object: a character called "Bob"; 

Current State: Bob is sitting down watching television in his lounge; 

Predetermined Change in Environmental Conditions: The telephone in the lounge rings; 

Event in Response to Sensing Predetermined Change: Bob gets up, walks over to the 

telephone, picks up the receiver and starts a conversation; 

New State: Bob is having a conversation on the telephone; 

Having described the present invention by reference to specific embodiments, it is to be 
appreciated that the described embodiments are exemplary only and are susceptible to 
modification and variation without departure from the spirit and scope of the invention as 
determined by the appended claims. It is also to be appreciated that the invention extends to 
any novel combination or subcombination disclosed herein, whether claimed or not. 
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Method and Apparatus for generating Moving Characters 

The present invention relates to a method and apparatus for generating moving 
characters. More particularly the invention relates, in preferred embodiments, to a 
method and apparatus (particularly a programmed personal computer) for creating 
real-time-rendered virtual actors (avatars). 

Techniques exist for creating photo-realistic avatars in eg arcade games and for 
cinema production but these normally rely either 1) on 3D laser scanning which is 
very expensive and produces models that are very demanding of processing power 
and memory and cannot be run on conventional personal computers (PC's), even 
those based on the Pentium chip, or 2) on Full Motion Video (FMV), which places 
filmed sequences in computer-playable formats. Such techniques have the following 
major disadvantages: 

a) the appearance of the characters in FMV is flat and non-immersive 

b) the colours in FMV are washed out 

c) the development cost of both FMV and laser scanning is high 

d) the game play is tedious and linear, restricted to pre-filmed or pre-rendered 
scenes 

e) the ROM demands of both are excessive. 

Alternatively, developers intercut occasional pre-rendered scenes using non-photo- 
realistic dummies, but 

a) being non-interactive, they inevitably interrupt gameplay 

b) they invariably lack realism 

c) very long development times are required. 

Furthermore this expedient does not avoid the inflexibility and excessive ROM 
demands of FMV. 

Currently no games developers use real-time photo-realistic 3D avatars in games for 
the PC platform. It is thought that the reason for this is that current 3D scanning 
technology is inappropriate for this application. In particular, existing whole body 
scanners produce models of sizes well in excess of 100,000 polygons, which cannot 
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therefore be rendered in real time on a PC. A reduction by a factor of 100 in the 
number of polygons is required for such real-time rendering and the polygon 
reduction tools which can be used for this purpose do not give acceptable results. 



An object of the present invention is to ovecomeor alleviate some or all of the 
above disadvantages. 

Accordingly in one aspect the invention provides a method of generating a moving 
character for a display, comprising the steps of i) providing: 

a) a three-dimensional representation of the structure of the character, the structure 
representation changing in real time to represent movement of the character, 

b) a three-dimensional representation of the surface of the character which is 
mapped onto said structure representation, and 

c) a two-dimensional representation of frequently changing portions of said surface, 



and 



20 ii) combining said representations a), b) and c) to generate a combined 
representation of the moving character. 

In another aspect the invention provides an apparatus for generating a moving 
character comprising: 

25 

i) memory means arranged to provide: 

a) a three-dimensional representation of the structure of the character, the structure 
representation changing in real time to represent movement of the character, 

b) a three-dimensional representation of the surface of the character which is 
mapped onto said structure representation, and 

c) a two-dimensional representation of frequently changing portions of said surface, 



35 and 



ii) processing means arranged to combine said representations a), b) and c) to 
generate a combined representation of the moving character. 
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In its preferred embodiments the present invention enables the creation of fully 
three-dimensional characters that are capable of moving and behaving in a 
convincingly human fashion, interacting with their environment on a data-driven 

^ level, supporting a variety of traditional photographic and cinematic techniques, 
such as moving from long-shots to close-ups without break or transition, and which 
provide lip-synchronisation and sufficient facial expression to enable a wide range of 
conversation and mood, all within the existing constraints of Teal-time rendering on 

10 PC's. 

In preferred embodiments in which the rapidly changing area of the character 
conveys speech and facial expression, emulation of a range of phonemes and a range 
of eye and forehead movements is provided for. It has been found that by 
15 subdividing the face into upper and lower areas and substituting texture maps 
accordingly, an acceptably wide range of expressions and mouth movements can be 
conveyed in an economical fashion. 



20 



In such embodiments the two-dimensional representations c) are for example of 
eye, mouth and preferably also forehead regions of the face. These are preferably 
photographic in origin. Consequently even slight, simple movements in the head 
convey a feeling of realism, as the viewer reads their own visual experience into the 
character in a way that cannot be done with any other illustrative technique. 

25 

Because the rapidly changing areas of the character are represented two- 
dimensionally (and therefore do not involve recalculation of position as the character 
changes orientation, as would be required in the case of a wire frame model in which 
changes in expression are generated by movements of polygons beneath the surface), 
30 less processing power, memory and bandwidth (if the image is transmitted eg over 
the Internet) is required. 

In one embodiment the production path for the avatars is as follows: 



35 



Photograph (in 35mm format) subjects head from front and side viewpoints, for 
geometry creation, placing a physical right-angled calibration target (a pre-marked 
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set square) horizontally within the photographed image 

Photograph subject head-on running through a fixed list of phonemes and facial 
expressions. 

Photograph subject's body from two viewpoints for geometry creation 

Using 3d plotting software (eg Wireframe Xpress, previously used for CAD 
10 purposes), plot each individual vertex and polygon by hand to describe the surface 
structure of the heads. 

Image re- touch all of the photographic texture-maps so that they synchronise in 
colour and lighting (eg using Aldus PhotoShop and various other graphics 
packages). 



15 
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Sub-divide geometry and apply texture-maps to appropriate areas. 

Create back and front segments for all limb sections, and retouch texture-maps 

Define inverse kinematics hierarchy and animate using captured motion ( eg using a 
combinatioaof Hock of Birds, Puppeteer, and 3D Studio) 

25 Feed into a suitable 3D engine (eg MUMM-E) which runs under, for instance, 
Argonaut's BRender real-time 3D engine. 

The result is believably human pseudo-video rendered on the fly at frame rates of 
12-16 frames a second on an ordinary pentium PC. 



30 
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A preferred embodiment of the invention will now be described by way of example 
only with reference to Figures 1 to 3 of the accompanying drawings, wherein: 

Figure 1 shows the generation of a structure representation mapped onto a surface 
representation of a human character shown A) in front view and B) in profile; 

Figure 2 shows the above character in front view with the frequently changing 
portions ( forehead, eyes and mouth) used for facial expression shown in different 



40 
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states, and 

Figure 3 is a diagrammatic representation of the apparatus used for the capture of 
motion of the body of the above character. 

The detailed production procedure for heads and faces is as follows: 
Stage 1: Photography 

I: Format: the camera negative format should be 35mm; so that the application 
can consistently calculate the camera position. 

II: Lens: the focal length of the lens should be ideally around 100m; this allows 
for perspective flattening effects to be made use of. 

Ill: Lighting: Lighting should be diffuse and evenly balanced across both sides of 
ihe face, most pleasing results have been obtained with warmer images. 
Photographing subjects under tungsten lighting using 100-400 ASA daylight 
balanced film has proven effective. 

IV: Triangulation and calibration: The subject is photographed with a calibration 
target S as shown in Figure 1A and Figure IB. The calibration target works well 
when it runs under the subject's chin, with about 200mm (8 inches) vertical 
separation. In order for the face to be mirrored correctly the front of the target must 
be parallel to the face. The Z axis of the target should move back over the subjects 
left shoulder. 

The resulting photographs 1 are subsequently used to generate a three-dimensional 
representation of the surface of the subject onto which is mapped a polygonal model 
2 of the underlying structure of the subject which in use is manipulated in three 
dimensions by the computer running the finished video game. This process will be 
explained in detail subsequently . 

V: Angles: best results are obtained with photographs taken at right angles, with 
profile (0 degrees) and full-face (90 degrees). The profile shot should cover the 
entire head. 
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VI: 



Mouth movements: the following movements and expressions should be 



included: 



5 



1: 


Eyes looking left 


2; 


Eyes looking right 


3: 


Sceptical look 


4: 


Laugh 


5: 


Wide eyes 


6: 


Raised eyebrows 


7: 


Mad/cross 


8: 


Sad look 


9: 


Half blink (sleepy) 


10: 


Full blink 


11: 


"Em" - Phonetic M sound - 


12: 


"eff" - Phonetic F sound - 


13: 


"Oh" - Phonetic O sound - 


14: 


"Yew" - Phonetic U sound - 


15: 


"En" - Phonetic N sound - 


16: 


"Arh" - Phonetic R sound - 


17: 


"Ae" - Phonetic A sound - 



25 With these movements care must be taken to ensure no extreme deformations of the 
jaw-line occur. Keeping the jaw lightly clenched whilst photographing the mouth 
movements reduces the risk of this. 

Typical expressions are illustrated in Figures 2A and 2B. The forehead, eyes and 

30 

mouth regions 3, 4 and 5 are recorded two-dimensionally and their different states 
are stored for use by the finished game program. Such use can involve appropriate 
combinations of states (eg frown, eyes left, lips compressed) according to the game 
play eg in response to commands from the user of the game. 

35 

Stage 2: Face digitisation 

I: Work with the profile (0 degree) photo in the left hand window, and the full- 
face (90 degree) in the right. 

40 
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II: Digitise the calibration target S first of all. A discrepancy can occur when 
minor mismatches between calibration data in the target settings window and pixel 
locations of the points in the images take place. The symptom is usually a 
misalignment between the calibration grid and one point of the triangle; often the 
triangle lies out of the calibration plane by around 15 to 25 degrees. The remedy is to 
selectively lengthen or shorten one of the dimensions in the target settings window, 
in quarter to half inch increments, until the triangle lies in the plane of the grid. 

Ill: The plotting of the 3d polygonal structure representation 2 on the 

photographic representation 1 of the surface is illustrated in Figures 1 A and IB. On 
the face, the first line to plot is that of the profile. This should extend from the curv e 
of the throat, up through the curv e of the chin, over the mouth, up under the nose, 
through to the bridge of the nose, and up the forehead to the hairline. This line 10 
(Figures 1A and IB) should be made up of 10-12 points, and be an open path. 



IV: This first profile line can then be taken into Excel to be straightened in the x- 
axis. The process for working with 3DP files in Excel is described later. 

20 

V: Next, the hairline should be followed down to the base of the left ear, and 
along the jaw. Again, 10-12 points are likely to be needed to keep this line smooth. 
This line should join the profile at the bottom of the chin. 

25 

VI: The first complex component is the eye. Plot this as a fiat plane, with a point 
in the centre - on the iris. This can be moved back and forth in the profile view to 
adjust for curvature. Plot open paths along the eyebrow, and down the eyelid. 
Connect these wi th faces. 

30 

VII: The second complex component is the nose, particularly the underside. Make 
frequent use of the perspective view tool to ensure that the line of the profile is not 
distorted. This is easy to do, and results in caricature-like geometry is in the final 
^ model. Be especially careful that the points under the nose in the two images relate. 

VIII: Build the forehead with a horizontal ribbon running across to the temple, just 
above the eyebrow. Use as few faces as possible, but ensure that the turn of the angle 
of the forehead is neither to swept or to sharp. 
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IX: Follow the pit of the eye, from eyebrow to nose, use this line as a basis for 
the rest of the front of the face. 

5 

X: Extend a ribbon across the mouth, probably no more than two or three points. 
Use this to create a plane from nose to chin; do not over-model the mouth, as the 
animated texture maps will not fit the geometry if you do. 

10 XII: Join the rest of the face with triangles. When finished, you will have a 
completed left hand side to the face. 



XIII: The next component is the side and back of head. The entire side and back 
can be extracted from the profile shot if careful. The first step is to start a new 
project, and then reverse the position of the photographs, so that the full-face is in 
the left window, and the profile is in the right. 

XIV: Begin by plotting from the centre of the hairline around to the base of the 
jaw, as close as possible to the ending point of the faces jawline. Finish this line with 
a point buried within the centre of the head. 

XV: Plot the outline of the ear. Use one, or if necessary two, points inside the ear 
to form faces in the same way as has been done with the eye. 

XVI: Build transverse ribbons through the hair, from the centre to the left edge of 
the head. 



XVII: For the back of the head, Drop .lines vertically down from the transverse 
crown-line towards the nape of the neck or the base of the hair. In order to establish 
which point is which when closing faces, use the select point tool (by clicking on a 
verier). 

XVIII: The final component is the neck and shoulders. The point at which you stop 
building out from the head will be a decision made in accordance with the structure 
for the rest of the model. It may be appropriate to limit this to the collar line, or 
extend the neck further into the throat and clavicle area. 



40 



BNSDOCID: <WO 9835320A1_I_> 



WO 98/35320 




PCT/GB98/00372 



XIX: Start a new project, with the profile in the left window now, and the fullface 
in the right band window. 

XX: The neck is essentially a cylinder, with sufficient modelling to fit into the 
base of the head and extend into the head. Three lines of 6 points running around the 
neck should suffice. Once completed, close faces as before. 



Stage 3: Mirroring 

10 I: It is now necessary to mirror the face, head/hair, and neck. The face needs 
more attention than the other two components, in that we need to move the texture 
map as well as mirror the geometry. 

II: Mirroring involves a dos-command line utility called, unsurprisingly, mirror. 
15 The syntax for this is very simply: mirror filename.3dp newfilename.3dp. Perform 
this function on each component. 

Ill: The mirroring process is not entirely stable. It is now necessary to take the 
new .3dp files into Excel or another spreadsheet (Excel is known to work) and 
recalculate certain vertex locations. The problem with mirror is that it will reflect 
geometry around the project centre point, not along the weld line of the model. In 
addition, mirror does not move the pixel location of the vertex co-ordinates in the 
texture map. This results in the texture map being mirrored as well as the geometry, 
rather than it picking up new co-ordinates from the new geometryis pixel location. 
The effect is somewhat counter-intuitive! 

IV: .3dp files are ASCII based text files, with a structure that can be defined as a 
space-delimited variable. Excel will import this structure, with some effort. 

V: Open the file in Excel. Specify the file type as a text file (with the extensions 
.txt, .csv and .prn). Add *.3dp to the file extensions list, and select the face file. 

VI: Excel will launch one of its "wizard" utilities; on the first page of this, select 
edelimitedi as being the file type. On the second, turn off "tab" and turn on "space". 
On the third, select "text" as being the data type. 
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VII: The file will open. Select all of columns A to J and change the column width 
to 20. Click on the "centre* icon. Often, an error message will occur saying 
"selection too large, continue without undo?" Answer yes. This file may now be 
worked on. 

VIII: A pproximately one third of the way down the file is the points database, with 
XYZ co-ordinates for the vertices. The first three point values will relate to the 
calibration target - ignore them, points 4 to n will be the values for original half of 
the face; look at the left hand column, which represents the x values. Note two 
things: first that the first half of the column has negative values, second that the first 
10 -12 points havethe same value. These are the first set of points defined for the 
profile. Make note of this value. Move down the column until the points turn from 
negative to positive. Thes are the values for the mirored half-face x coordinates. 
These values must be changed. 

IX: The formula for moving these is =SUM(value)-(2*.x), where x is the value 
from your profile line. Change all the values for the second half of the column using 
this. 

X: At the top of the file is the points database. This gives the values for the 
equivalent location in x-y pixel coordinates for the vertices in leftand .right pictures. 
There are four columns (apart from the numbering of the points) They read x-left 
hand y-left hand x-right hand y right hand. The column to be concerned with is the 
x-right hand. Again, locate the mirror portion by noting the discrepancies in values, 
and identify the number given to the weld line. The same process as with the points 
database needs to be performed, but in this case the formula is SUM(y-value+y), 
where y is the vlue in pixels from the welding line. 

XI: Do the above for both the neck and hair/head, but do not mirror the pixel 
database - symmetrical hairstyles are easier corrected in 3d Studio. 

XII: The next step is to convert the 3dp files to 3d Studio files and assign texture 
vertices. Again a command line tool, called convert, is used. The syntax is convert 
filename.3dp -r name of right-hand photograph.bmp matx.tga . 

XII: It may well be necessary to work on the texturemap files; the hair may need 
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to be cloned outwards, the same for the neck in the neck texture, and also clean hair 
off the forehead by cloning. 

XIII: Take files into 3ds t join and move vertices where necessary. 



Stage 4: Lip Synching 

I: Using the basic fullface image, move to Animator Pro. Load all of the 
subsequent facial movements as a flic file. 

II: Cut out the active area of the face, i.e. around the eyes and down to the mouth, 
taking care to include areas of skin that might also move. 

Ill: Add these cut-outs in the correct positions down onto the basic face of the 
subject, using the original fullface shot 

IV: Cut out the areas of the photograph (3, 4 or 5 - Figures 2A and 2B) that will 
move from frame to frame. For example cut out one of the moving eyes and paste it 
down onto the original area of the image. Also select some of the surrounding skin 
and using the C (clip) and L (lasso) tools. Pick up parts of the skin using the C (clip) 
and M (move) tools. Use the " apostrophe mark to paste these into the final flic, 

V: The flic file should then contain all of the images from the original photoshoot 
of seventeen expressions, and if necessary any in-between frames that are required to 
smooth the facial movement. From this flic file, a render in 3d Studio can ensure that 
none of the fluid facial movements exceed the limits of the static facial geometry. 
For final export into Brender or Rendermorphics, individual frames can be 
generated. 

VI: Apply the final animated file (saved in .eel format) to the facial geometry in 
3ds. Construct a hierarchy that links the face and the head/hair, and place its pivot 
point at the nape of the neck. This allows for convincing head movement, combined 
with photographically realistic facial expressions. Adjust imperfections accordingly. 

The body of the character can be animated and combined with the head by standard 
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techniques. One way of capturing motion of the body is illustrated in Figure 3 
wherein magnets are mounted at the joints of the subject's body 8 and their position 
detected by an electromagnetic positon sensor 8 of standard type. 

The avatar derived from such a technique can be controlled and displayed by a 
computer game. 

The invention extends to any novel combination or sub-combination disclosed 
herein. In particular any photographic representation of the rapidly changing areas of 
the character, whether or not two-dimensional, may be employed. 
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Claims 



1. A method of generating a moving character for a display, comprising the steps of 
i) providing: 

a) a three-dimensional representation of the structure of the character, the structure 
representation changing in real time to represent movement of the character, 

b) a three-dimensional representadon of the surface of the character which is 
mapped onto said structure representation, and 

c) a two-dimensional representation of frequently changing portions of said surface, 



ii) combining said representations a), b) and c) to generate a combined 
representation of the moving character 

2. A method as claimed in claim 1 wherein said combined representation is 
generated in real time. 

3. A method as claimed in claim 1 or claim 2 wherein said combined representation 
is displayed interactively in response to commands from a user. 

4. A method as claimed in any preceding claim wherein two or more such two- 
dimensional representations c) of different frequently-changing changing portions of 
said surface are generated and are combined with said representations a) and b). 

5. A method as claimed in any preceding claim wherein the two-<limensional 
representation(s) c) is/are of portion(s) of facial expressions. 

6. A method as claimed in any preceding claim wherein said three-dimensional 
surface representation b) is derived from a photograph. 

7. A method as claimed in any preceding claim wherein said three-dimensional 
structure representation a) comprises a multiplicity of polygons. 

8- A method as claimed in any preceding claim wherein said combining step ii) is 
carried out by a program running in a PC and the character is displayed in real time 



and 
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on the display of the PC. 
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9. An apparatus for generating a moving character comprising: 

i) memory means arranged to provide: 

a) a three-dimensional representation of the structure of the character, the structure 
representation changing in real lime to represent movement of the character, 

b) a three-dimensional representation of the surface of the character which is 
mapped onto said structure representation, and 

c) a two-dimensional representation of frequently changing portions of said surface, 
and 

ii) processing means arranged to combine said representations a), b) and c) to 
generate a combined representation of the moving character. 

10. An apparatus as claimed in claim 9 wherein said processing means is arranged 
to generate said combined representation in real time. 

1 1. An apparatus as claimed in claim 9 or claim 10 wherein said processing means 
is providedwith user-controlled input means and is arranged to display said 
combined representation interactively in response to commands from a user. 

12. An apparatus as claimed in any of claims 9 to 1 1 which is arranged to combine 
two or more such two-dimensional representations c) of different frequently- 
changing changing portions of said surface with said representations a) and b). 

13. An apparatus as claimed in any of claims 9 to 12 wherein the two-dimensional 
represenlation(s) c) is/are of portion(s) of facial expressions. 

14. An apparatus as claimed in any of claims 9 to 13 wherein said three- 
dimensional surface representation b) is derived from a photograph. 

15. An apparatus as claimed in any preceding claim Wherein said three-dimensional 
structure representation a) comprises coordinates defining a multiplicity of polygons. 
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16. An apparatus as claimed in any of claims 9 to 15 which is a PC arranged to 
display the character in real time on its display. 



17. An avatar comprising: 

a) an animated three-dimensional structure representation f 

b) a three-dimensional surface representation which is mapped onto said structure 
representation, and 

c) a two-dimensional representation of frequently changing portions of the surface 
of the avatar. 

18. A method of generating a moving character substantially as described 
hereinbabove with reference to Figures 1 to 3 of the accompanying drawings. 
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Abstract 

Avatar generation 

An avatar is generated by 

i) providing: 

a) a three-dimensional representation (2) of the structure of the character, the 
structure representation changing in real time to represent movement of the 
character, 

b) a three-dimensional representation of the surface (1) of the character which is 
mapped onto said structure representation, and 

c) a two-dimensional representation of frequently changing portions of said surface, 
eg the portions used to generate facial expressions 

and 

ii) combining said representations a), b) and c) to generate a combined 
representation of the moving character. Because the frequently changing portions are 
represented only two-dimensionally, less processing power and ROM is needed to 
display the avatar. 
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Figure 1 
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Claims 



1 . A system for managing animation, the system comprising a processor arranged to 
process a dynamically evolving data set including object data and event data, said 
object data including character data representing avatars, sequencing means arranged 
to read data from and write data to said dynamically evolving data set in a 
programmed sequence, said sequence of reading and writing taking place within a 
frame period and said written data being generated from said read data by stored 
interactions between said object data and event data, and output means arranged to 
generate from said dynamically evolving data set a displayable animated output. 

2. A system according to claim 1, wherein said data set includes stored events, each 
event comprising a set of one or more preconditions associated with a set of one or 
more results, and an events manager controlled by said sequencing means is 
arranged to detect said set of preconditions and, on detection of said set of 
preconditions, to generate said set of results, said output means being responsive to 
said results to generate said displayable animated output. 

3. A system according to claim 2, wherein the data set includes stored objects, each 
object having one or more events associated therewith, and said events manager is 
arranged to poll said objects to establish which events are ready for firing and which 
objects are associated with the events ready for firing. 

4. A system according to claim 2, wherein the data set includes stored objects, each 
object having one or more events associated therewith, and said events manager 
utilises a set of state machine maps to determine which objects have events ready for 
firing and how the object is affected by the firing of those events, each map 
describing all the possible states a particular object can reside in, the events involved 
in the movement from one state to another and the predetermined conditions 
required to trigger said movement from state to state. 

5. A system according to any of claims 2 to 4, further comprising animation routines 
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arranged to be triggered by said generated sets of results, said output means being 
arranged to run said triggered animation routines to generate said displayable 
animated output. 

6. A system according to any of claims 2 to 5, further comprising rendering means 
arranged to render a scene composed of objects associated with said generated sets 
of results. 

7. A system according to any preceding claim, wherein the object data represent a 
three-dimensional world. 

8. A system according to any preceding claim, comprising conversation means 
arranged to generate and respond to dialogue between avatars, said output means 
being responsive to the output of said conversation means. 

9. A system according to claim 8, wherein said conversation means is arranged to 
process dialogue by traversing a conversational net having linked nodes, the 
branching from each node representing the possible continuations of the dialogue. 



10. A system according to any preceding claim, wherein said object data is in the 
form of hierarchical trees of object types, each object type having an associated set 
of parameters and object types at a higher level in the tree having parameters generic 
to object types to which they are linked at a lower level, object types being 
distinguished by additional parameters appropriate to their type. 

11. A system according to claim 10 wherein said object types include a generic 
Aartefact@ object type which is linked at a lower level in the tree with a 
Acontainer@ object type and by a Acamera® object type. 



12. A system according to claim 10 or 11, wherein said object types include a 
generic Acontainer@ object type which is linked at a lower level in the tree with a 
Aroom@ object type and by a Aperson@ object type. 
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13. A system according to any preceding claim, wherein said object data includes 
data representing groups of associated objects which are physically associated with 
one another in the animated output, the data representing such groups being a 
hierarchical tree wherein objects at a higher level in the tree, when subjected to said 
event data cause objects linked thereto at a lower level in the tree to share at least 
some aspects of their behaviour. 

14. A system according to any preceding claim, comprising a user interface arranged 
to enable a user to create said object and event data and thereby generate an 
animation sequence. 

15. A system according to claim 14, comprising at least one animation engine 
responsive to instructions from said user interface to generate animation subroutines 
and to incorporate them in said data structure. 

16. A system according to claim 15, wherein said animation engine includes an input 
arranged to receive captured motion from motion capture apparatus. 

17. A system according to claim 15 or 16, further comprising a lip synchronisation 
engine coupled to said user interface and arranged to generate lip synchronisation 
data from input speech, said animation engine being responsive to said lip 
synchronisation data to synchronise lip movement of the avatars with their speech. 

18. A system according to claim 17, wherein stored sampled speech is accessible 
from said user interface and is arranged to be played in response to keyboard entry 
of the words or phrases corresponding to the speech. 

19. A system according to any preceding claim, wherein said dynamically evolving 
data set has an input enabling a user to interact with said avatars by modifying said 
data set. 
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20. A system according to any preceding claim, wherein said system is arranged to 
process said dynamically evolving data set to provide a real-time animated output in 
a format which is suitable for carrying out a non-linear interactive storyline. 

21. A system according to claim 20, wherein said system is arranged also to process 
said dynamically evolving data set to provide an animated output in a format which 
can be used for carrying out a linear non-interactive storyline. 

22. A method of generating an animation sequence, the method comprising the steps 
of generating a dynamically evolving data set including object data and event data, 
said object data including character data representing avatars, reading data from and 
writing data to said dynamically evolving data set in a programmed sequence, said 
sequence of reading and writing taking place within a frame period and said written 
data being generated from said read data by stored interactions between said object 
data and event data, and generating a displayable animated output from said 
dynamically evolving data set. 

23. A method as claimed in claim 22, wherein said data set is initially generated at a 
user interface as defined in any of claims 14 to 16. 

24. A method as claimed in claim 22 or 23, wherein the animation sequence is 
generated by means of a system as claimed in any of claims 2 to 13. 

25. A method as claimed in any of claims 22 to 24, wherein a program containing 
stored input instructions in a high level language and governing the animation 
sequence is run and is arranged to modify said dynamically evolving data set. 

26. A computer programmed with a computer game containing an animation 
sequence generated by a method as claimed in any of claims 22 to 25. 
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